New Frontiers in Interactive Multimodal Communication

This essay describes two emergent phenomena related to multimodality in digital communications. The first phenomenon is interactive multimodal platforms—Web 2.0 platforms that support a convergence of channels or “modes” (text, audio, video, images) for user-to-user communication. The second is robot-mediated communication—humanhuman communication in which at least one party is telepresent through voice, video, and motion in physical space via a remotely controlled robot. At first blush, these two phenomena may appear unrelated—websites are on the Internet, whereas robots are physical, mechanical objects; web interaction is persistent and often asynchronous, whereas robot-mediated interaction takes place in real time and does not leave a verbal trace; and so forth. At the same time, both technologies mediate human-to-human communication, support social as well as task-related interaction, and involve multiple modes or channels. More generally, both can be situated under the broad rubric of multimodal computer-mediated communication (CMC), as represented in Figure 1.